Electronic apparatus, control method thereof, computer program, and computer-readable recording medium

ABSTRACT

Provided herein is a control method of an electronic apparatus. The control method of an electronic apparatus includes: detecting a crosswalk from an image data photographed in a camera during a period in which a vehicle is operated; generating an object indicating the detected crosswalk; and outputting the generated object through augmented reality.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 15/814,150 filed on Nov. 15, 2017, which is acontinuation application of U.S. patent application Ser. No. 14/955,762,filed on Dec. 1, 2015, and claims priority to Korean Patent ApplicationNos. 10-2014-0170051 and 10-2015-0035744 which were filed on Dec. 1,2014 and Mar. 16, 2015, respectively, with the Korean IntellectualProperty Office, the entire disclosures of which are incorporated hereinby reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an electronic apparatus, a controlmethod thereof, a computer program, and a computer-readable recordingmedium, and more particularly, to an electronic apparatus capable ofperforming driving related guidance on a user on augmented reality, acontrol method thereof, a computer program, and a computer-readablerecording medium.

2. Description of the Related Art

It is most important to safely drive a vehicle and prevent a trafficaccident at the time of driving the vehicle. To this end, various assistapparatuses performing an attitude control of the vehicle, a functioncontrol of components of the vehicle, and the like, and safetyapparatuses such as a safety belt, an air bag, and the like, have beenmounted in the vehicle.

In addition, recently, apparatuses such as a black box, and the like,positioned in the vehicle and storing a driving image of the vehicle anddata transmitted from various sensors therein to find out a cause of anaccident of the vehicle at the time of occurrence of the accident havebeen provided in the vehicle. Portable terminals such as a smart phoneand a tablet personal computer (PC) in which a black box application, anavigation application, or the like, may be mounted have been used asthe apparatuses for a vehicle as described above.

However, currently, utilization of the driving image is low in theapparatuses for a vehicle as described above. In more detail, currently,even though the driving image of the vehicle is obtained through avision sensor such as a camera mounted in the vehicle, an electronicapparatus of the vehicle has simply displayed or transmitted the drivingimage or has generated only simple surrounding notification informationsuch as whether or not the vehicle has departed from a lane, or thelike.

In addition, a head-up display (HUD) or an augmented reality interfacehas also been suggested as an electronic apparatus of the vehicle thathas newly emerged currently. However, in the HUD and the augmentedreality interface, the driving image of the vehicle has been utilized tobe simply displayed or generate the simple surrounding notificationinformation.

SUMMARY

An object of the present invention is to provide an electronic apparatuscapable of generating an object indicating a crosswalk in a situationsuch as a case in which a vehicle stops at an intersection while beingdriven on a road, or the like, and outputting the generating objectthrough augmented reality, a control method thereof, a computer program,and a computer-readable recording medium.

According to an exemplary embodiment of the present invention, there isprovided a control method of an electronic apparatus, including:detecting a crosswalk from an image data photographed in a camera duringa period in which a vehicle is operated; generating an object indicatingthe detected crosswalk; and outputting the generated object throughaugmented reality.

The control method of an electronic apparatus may further includedetermining whether or not the vehicle is in a stopped state, wherein inthe generating, a first object for allowing a driver to recognize thatthe crosswalk is positioned in front of the vehicle is generated when itis determined that the vehicle is in the stopped state.

The control method of an electronic apparatus may further includedetermining signal kind information using an image data of a signalregion portion of a signal lamp in the image data, wherein in thegenerating, a first object for allowing a driver to recognize that thecrosswalk is positioned in front of the vehicle is generated when thevehicle is maintained in a stopped state in a state in which the signalkind information is a stop signal, and a second object for warning thedriver that the crosswalk is positioned in front of the vehicle isgenerated when the vehicle starts in the state in which the signal kindinformation is the stop signal.

The first and second objects may be distinguished from each other bydifferent colors.

The first and second objects may be implemented in a form including analpha channel related to transparency of a color, and the first andsecond objects may include transparent regions depending on the alphachannel.

The control method of an electronic apparatus may further include:determining whether or not a pedestrian is present on the crosswalkusing the photographed image data; and generating an object indicatingwhether or not the pedestrian is present.

The control method of an electronic apparatus may further includeperforming a control so as not to perform front vehicle start guidancein the case in which a front vehicle of the vehicle starts in a state inwhich the pedestrian is present on the crosswalk.

The control method of an electronic apparatus may further includeperforming a control to perform guidance for warning a driver that thepedestrian is present on the crosswalk in the case in which the vehiclestarts in a state in which the pedestrian is present on the crosswalk.

Expression positions of the first and second objects may be regions inwhich the crosswalk is positioned on the augmented reality.

The outputting may include: performing calibration on the camera tocalculate camera parameters; generating a virtual 3D space for aphotographed image of the camera on the basis of the camera parameters;and positioning the generated object on the virtual 3D space.

According to another exemplary embodiment of the present invention,there is provided an electronic apparatus including: a display unitdisplaying a screen; an object generating unit generating an objectindicating a crosswalk when the crosswalk is detected from an image dataphotographed in a camera during a period in which a vehicle is operated;and a control unit controlling the display unit to output the generatedobject through augmented reality.

The control unit may determine whether or not the vehicle is in astopped state, and control the object generating unit to generate afirst object for allowing a driver to recognize that the crosswalk ispositioned in front of the vehicle when it is determined that thevehicle is in the stopped state.

The control unit may determine signal kind information using an imagedata of a signal region portion of a signal lamp in the image data, andcontrol the object generating unit to generate a first object forallowing a driver to recognize that the crosswalk is positioned in frontof the vehicle when the vehicle is maintained in a stopped state in astate in which the signal kind information is a stop signal and generatea second object for warning the driver that the crosswalk is positionedin front of the vehicle when the vehicle starts in the state in whichthe signal kind information is the stop signal.

The first and second objects may be distinguished from each other bydifferent colors.

The first and second objects may be implemented in a form including analpha channel related to transparency of a color, and the first andsecond objects may include transparent regions depending on the alphachannel.

The control unit may determine whether or not a pedestrian is present onthe crosswalk using the photographed image data, and control the objectgenerating unit to generate an object indicating whether or not thepedestrian is present.

The control unit may perform a control so as not to perform frontvehicle start guidance in the case in which a front vehicle of thevehicle starts in a state in which the pedestrian is present on thecrosswalk.

The control unit may perform a control to perform guidance for warning adriver that the pedestrian is present on the crosswalk in the case inwhich the vehicle starts in a state in which the pedestrian is presenton the crosswalk.

Expression positions of the first and second objects may be regions inwhich the crosswalk is positioned on the augmented reality.

The control unit may perform a control to perform calibration on thecamera to calculate camera parameters, generate a virtual 3D space for aphotographed image of the camera on the basis of the camera parameters,and position the generated object on the virtual 3D space.

According to still another exemplary embodiment of the presentinvention, there is provided a computer program stored in a recordingmedium coupled to an electronic apparatus to execute the followingsteps: detecting a crosswalk from an image data photographed in a cameraduring a period in which a vehicle is operated; generating an objectindicating the detected crosswalk; and outputting the generated objectthrough augmented reality.

According to yet still another exemplary embodiment of the presentinvention, there is provided a computer-readable recording medium inwhich a computer program for executing a control method of an electronicapparatus is stored, wherein the control method of an electronicapparatus includes: detecting a crosswalk from an image dataphotographed in a camera during a period in which a vehicle is operated;generating an object indicating the detected crosswalk; and outputtingthe generated object through augmented reality.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an electronic apparatus accordingto an exemplary embodiment of the present invention.

FIG. 2 is a block diagram illustrating an augmented reality providingunit according to an exemplary embodiment of the present invention indetail.

FIG. 3 is a view for describing a system network connected to anelectronic apparatus according to an exemplary embodiment of the presentinvention.

FIG. 4 is a flow chart schematically illustrating a control method of anelectronic apparatus according to an exemplary embodiment of the presentinvention.

FIG. 5 is a flow chart illustrating a control method of an electronicapparatus according to an exemplary embodiment of the present inventionin detail.

FIG. 6 is a flow chart illustrating a control method of an electronicapparatus according to another exemplary embodiment of the presentinvention in detail.

FIG. 7 is a flow chart illustrating a control method of an electronicapparatus according to still another exemplary embodiment of the presentinvention in detail.

FIGS. 8A and 8B are views illustrating augmented reality screensexpressing a crosswalk object according to an exemplary embodiment ofthe present invention.

FIG. 9 is a view illustrating a texture image of the crosswalk objectaccording to an exemplary embodiment of the present invention.

FIG. 10 is a view illustrating an augmented reality screen expressing anotification object of a pedestrian according to an exemplary embodimentof the present invention.

FIG. 11 is a view illustrating an implementation in the case in which acamera and an electronic apparatus according to an exemplary embodimentof the present invention are separated from each other.

FIG. 12 is a view illustrating an implementation in the case in which acamera and an electronic apparatus according to an exemplary embodimentof the present invention are integrated with each other.

FIG. 13 is a view illustrating an implementation using a head-up display(HUD) and an electronic apparatus according to an exemplary embodimentof the present invention.

DETAILED DESCRIPTION

The following description illustrates only a principle of the presentinvention. Therefore, those skilled in the art may implement theprinciple of the present invention and invent various apparatusesincluded in the spirit and scope of the present invention although notclearly described or illustrated in the present specification. Inaddition, it is to be understood that all conditional terms andexemplary embodiments mentioned in the present specification areobviously intended only to allow those skilled in the art to understanda concept of the present invention in principle, and the presentinvention is not limited to exemplary embodiments and statesparticularly mentioned as such.

Further, it is to be understood that all detailed descriptionsmentioning specific exemplary embodiments of the present invention aswell as principles, aspects, and exemplary embodiments of the presentinvention are intended to include structural and functional equivalencesthereof. Further, it is to be understood that these equivalences includean equivalence that will be developed in the future as well as anequivalence that is currently well-known, that is, all devices inventedso as to perform the same function regardless of a structure.

Therefore, it is to be understood that, for example, block diagrams ofthe present specification illustrate a conceptual aspect of anillustrative circuit for embodying a principle of the present invention.Similarly, it is to be understood that all flow charts, state transitiondiagrams, pseudo-codes, and the like, illustrate various processes thatmay be tangibly embodied in a computer-readable medium and that areexecuted by computers or processors regardless of whether or not thecomputers or the processors are clearly illustrated.

Functions of various devices including processors or functional blocksrepresented as concepts similar to the processors and shown in theaccompanying drawings may be provided by using hardware havingcapability to execute appropriate software as well as dedicatedhardware. When the functions are provided by the processors, they may beprovided by a single dedicated processor, a single shared processor, ora plurality of individual processors, and some of them may be shared.

In addition, terms mentioned as a processor, a control, or a conceptsimilar to the processor or the control should not be interpreted toexclusively cite hardware having capability to execute software, butshould be interpreted to implicitly include digital signal processor(DSP) hardware and a read only memory (ROM), a random access memory(RAM), and a non-volatile memory for storing software without beinglimited thereto. The above-mentioned terms may also include well-knownother hardware.

In the claims of the present specification, components represented asmeans for performing functions mentioned in a detailed description areintended to include all methods of performing functions including alltypes of software including, for example, a combination of circuitdevices performing these functions, firmware/micro codes, or the like,and are coupled to appropriate circuits for executing the software so asto execute these functions. It is to be understood that since functionsprovided by variously mentioned means are combined with each other andare combined with a scheme demanded by the claims in the presentinvention defined by the claims, any means capable of providing thesefunctions are equivalent to means recognized from the presentspecification.

The above-mentioned objects, features, and advantages will become moreobvious from the following detailed description associated with theaccompanying drawings. Therefore, those skilled in the art to which thepresent invention pertains may easily practice a technical idea of thepresent invention. Further, in describing the present invention, in thecase in which it is determined that a detailed description of awell-known technology associated with the present invention mayunnecessarily make the gist of the present invention unclear, it will beomitted.

Hereinafter, various exemplary embodiments of the present invention willbe described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram illustrating an electronic apparatus accordingto an exemplary embodiment of the present invention. Referring to FIG.1, the electronic apparatus 100 is configured to include all or some ofa storing unit 110, an input unit 120, an output unit 130, a crosswalkdetecting unit 140, a signal kind information determining unit 150, anoperated state determining unit 155, an augmented reality providing unit160, a control unit 170, a communicating unit 180, a sensing unit 190,and a power supply unit 195.

Here, the electronic apparatus 100 may be implemented by variousapparatuses such as a smart phone, a tablet computer, a laptop computer,a personal digital assistant (PDA), a portable multimedia player (PMP),a smart glasses, a project glasses, a navigation apparatus, a black box,and the like, that may provide driving related guidance to a driver of avehicle that is in an operated state, and may be provided in thevehicle.

Here, the operated state of the vehicle may include various states inwhich the vehicle is being driven by the driver, such as a stopped stateof the vehicle, a driven state of the vehicle, a parked state of thevehicle, and the like.

The driving related guidance may include various kinds of guidance forassisting in driving of the driver of the vehicle, such as routeguidance, lane departure guidance, front vehicle start guidance, signallamp change guidance, front vehicle collision preventing guidance, lanechange guidance, lane guidance, and the like.

Here, the route guidance may include augmented reality route guidanceperforming route guidance by combining various information such as aposition, a direction, and the like, of a user with an image obtained byphotographing the front of the vehicle that is being operated andtwo-dimensional (2D) or three-dimensional (3D) route guidance performingroute guidance by combining various information such as a position, adirection, and the like, of a user with a 2D or 3D map data. Here, theroute guidance may be interpreted as a concept including route guidancein the case in which the user walks or runs and moves as well as in thecase in which the user gets in the vehicle and then drives the vehicle.

In addition, the lane departure guidance may be to guide whether or notthe vehicle that is being driven has departed from a lane.

In addition, the front vehicle start guidance may be to guide whether ornot a vehicle positioned in front of a vehicle that is being stopped hasstarted.

In addition, the signal lamp change guidance may be to guide whether ornot a signal lamp positioned in front of a vehicle that is being stoppedhas been changed. As an example, the signal lamp change guidance may beto guide that a state of the signal lamp is changed from a red lampindicating a stop signal into a green lamp indicating a start signal.

In addition, the front vehicle collision preventing guidance may be toguide that a distance between a vehicle that is being stopped or drivenand a vehicle positioned in front of the vehicle is within apredetermined distance in order to prevent collision between theabove-mentioned vehicles when the distance between the vehicle that isbeing stopped or driven and the vehicle positioned in front of thevehicle is within the predetermined distance.

In addition, the lane change guidance may be to guide a change from alane in which a vehicle is positioned into another lane in order toguide a route up to a destination.

In addition, the lane guidance may be to guide a lane in which a vehicleis currently positioned.

A driving related image such as a front image of the vehicle enablingprovision of various kinds of guidance may be photographed by a cameramounted in the vehicle. Here, the camera may be a camera formedintegrally with the electronic apparatus 100 mounted in the vehicle andphotographing the front of the vehicle. In this case, the camera may beformed integrally with a smart phone, a navigation apparatus, or a blackbox, and the electronic apparatus 100 may receive the image photographedby the camera formed integrally therewith.

As another example, the camera may be a camera mounted in the vehicleseparately from the electronic apparatus 100 and photographing the frontof the vehicle. In this case, the camera may be a black box separatelymounted toward the front of the vehicle, and the electronic apparatus100 may receive a photographed image through wired/wirelesscommunication with the separately mounted black box or receive thephotographed image when a storage medium storing the photographed imageof the black box therein is inserted into the electronic apparatus 100.

Hereinafter, the electronic apparatus 100 according to an exemplaryembodiment of the present invention will be described in more detail onthe basis of the above-mentioned content.

The storing unit 110 serves to store various data and applicationsrequired for an operation of the electronic apparatus 100 therein.Particularly, the storing unit 110 may store data required for theoperation of the electronic apparatus 100, for example, an operatingsystem (OS), a route search application, a map data, and the like,therein. In addition, the storing unit 110 may store data generated bythe operation of the electronic apparatus 100, for example, a searchedroute data, a received image, and the like, therein.

Here, the storing unit 110 may be implemented by a detachable type ofstoring device such as a universal serial bus (USB) memory, or the like,as well as an embedded type of storing device such as a random accessmemory (RAM), a flash memory, a read only memory (ROM), an erasableprogrammable ROM (EPROM), an electronically erasable and programmableROM (EEPROM), a register, a hard disk, a removable disk, a memory card,a universal subscriber identity module (USIM), or the like.

The input unit 120 serves to convert a physical input from the outsideof the electronic apparatus 100 into a specific electrical signal. Here,the input unit 120 may include all or some of a user input unit 121 anda microphone unit 123.

The user input unit 121 may receive a user input such as a touch, a pushoperation, or the like. Here, the user input unit 121 may be implementedusing at least one of a form of various buttons, a touch sensorreceiving a touch input, and a proximity sensor receiving an approachingmotion.

The microphone unit 123 may receive a speech of the user and a soundgenerated in the inside and the outside of the vehicle.

The output unit 130 is a unit outputting data of the electronicapparatus 100. Here, the output unit 130 may include all or some of adisplay unit 131 and an audio output unit 133.

The display unit 131 is a unit outputting data that may be visuallyrecognized in the electronic apparatus 100. The display unit 131 may beimplemented by a display unit provided on a front surface of a housingof the electronic apparatus 100. In addition, the display unit 131 maybe formed integrally with the electronic apparatus 100 and output visualrecognition data, or may be installed separately from the electronicapparatus 100 like a head-up display (HUD) and output visual recognitiondata.

The audio output unit 133 is a unit outputting data that may beauditorily recognized in the electronic apparatus 100. The audio outputunit 133 may be implemented by a speaker representing a data that is tobe reported to the user of the electronic apparatus 100 as a sound.

The communicating unit 180 may be provided in order for the electronicapparatus 100 to communicate with other devices. The communicating unit180 may include all or some of a position data unit 181, a wirelessInternet unit 183, a broadcasting transmitting and receiving unit 185, amobile communicating unit 186, a short range communicating unit 187, anda wired communicating unit 189.

The position data unit 181 is a device obtaining position data through aglobal navigation satellite system (GNSS). The GNSS means a navigationsystem that may calculate a position of a receiving terminal using aradio signal received from an artificial satellite. A detailed exampleof the GNSS may include a global positioning system (GPS), a Galileosystem, a global orbiting navigational satellite system (GLONASS), aCOMPASS, an Indian regional navigational satellite system (IRNSS), aquasi-zenith satellite system (QZSS), and the like, depending on anoperating subject of the GNSS. The position data unit 181 of theelectronic apparatus 100 according to an exemplary embodiment of thepresent invention may obtain position data by receiving GNSS signalsserved in a zone in which the electronic apparatus 100 is used.

The wireless Internet unit 183 is a device accessing the wirelessInternet to obtain or transmit data. The wireless Internet that may beaccessed through the wireless Internet unit 183 may be a wireless localarea network (WLAN), a wireless broadband (Wibro), a worldinteroperability for microwave access (Wimax), a high speed downlinkpacket access (HSDPA), or the like.

The broadcasting transmitting and receiving unit 185 is a devicetransmitting and receiving broadcasting signals through variousbroadcasting systems. The broadcasting system that may transmit andreceive the broadcasting signals through the broadcasting transmittingand receiving unit 185 may be a digital multimedia broadcastingterrestrial (DMBT), digital multimedia broadcasting satellite (DMBS), amedia forward link only (MediaFLO), a digital video broadcast handheld(DVBH), an integrated services digital broadcast terrestrial (ISDBT), orthe like. The broadcasting signals transmitted and received through thebroadcasting transmitting and receiving unit 185 may include a trafficdata, a living data, and the like.

The mobile communicating unit 186 may access a mobile communicationnetwork to perform communication depending on various mobilecommunication protocols such as 3rd generation (3G), 3rd generationpartnership project (3GPP), long term evolution (LTE), and the like.

The short range communicating unit 187 is a device for short rangecommunication. The short range communicating unit 187 may performcommunication through Bluetooth, radio frequency identification (RFID),infrared data association (IrDA), ultra wideband (UWB), ZigBee, nearfield communication (NFC), wireless-fidelity (Wi-Fi), or the like, asdescribed above.

The wired communicating unit 189 is an interface device that may connectthe electronic apparatus 100 to another device in a wired scheme. Thewired communicating unit 189 may be a USB module that may performcommunication through a USB port.

The communicating unit 180 may communicate with another device using atleast one of the position data unit 181, the wireless Internet unit 183,the broadcasting transmitting and receiving unit 185, the mobilecommunicating unit 186, the short range communicating unit 187, and thewired communicating unit 189.

As an example, in the case in which the electronic apparatus 100 doesnot include a camera function, an image photographed by a camera for avehicle such as a black box, or the like, may be received using at leastone of the short range communicating unit 187 and the wiredcommunicating unit 189.

As another example, in the case in which the electronic apparatuscommunicates with a plurality of devices, the electronic apparatus maycommunicate with any one of the plurality of devices through the shortrange communicating unit 187, and communicate with another device of theplurality of devices through the wired communicating unit 119.

The sensing unit 190 is a unit that may sense a current state of theelectronic apparatus 100. The sensing unit 190 may include all or someof a motion sensing unit 191 and a light sensing unit 193.

The motion sensing unit 191 may sense motion of the electronic apparatus100 on a 3D space. The motion sensing unit 191 may include a tri-axialterrestrial magnetism sensor and a tri-axial acceleration sensor. Motiondata obtained through the motion sensing unit 191 may be combined withthe position data obtained through the position data unit 181 to moreaccurately calculate a trajectory of the vehicle to which the electronicapparatus 100 is attached.

The light sensing unit 193 is a device measuring surrounding illuminanceof the electronic apparatus 100. Brightness of the display unit 131 maybe changed so as to correspond to surrounding brightness usingilluminance data obtained through the light sensing unit 193.

The power supply unit 195 is a device supplying power required for anoperation of the electronic apparatus 100 or an operation of anotherdevice connected to the electronic apparatus 100. The power supply unit195 may be a device receiving power from an external power supply suchas a battery embedded in the electronic apparatus 100, the vehicle, orthe like. Alternatively, the power supply unit 195 may be implemented bythe wired communicating unit 189 or a device receiving power in awireless scheme depending on a scheme in which it receives the power.

The crosswalk detecting unit 140 may detect a crosswalk from an imagedata photographed in the camera. In detail, the crosswalk detecting unit140 may decide an interest region including a crosswalk in the imagedata using a vanishing point of the photographed image. Here, thevanishing point may be decided by extracting lanes from a photographeddata photographed in the camera during a period in which the vehicle isoperated and extending the extracted lanes to extract an intersectiontherebetween. In addition, since the crosswalk is formed on a road in alower end region of the vanishing point, the crosswalk detecting unit140 may determine that the lower end region of the vanishing point is aninterest region.

Meanwhile, the crosswalk detecting unit 140 may perform image processingon an image data on the decided interest region to determine whether ornot the crosswalk is positioned in the interest region.

However, unlike the example described above, according to anotherexemplary embodiment of the present invention, the crosswalk detectingunit 140 may detect the crosswalk without using the image dataphotographed in the camera. As an example, the crosswalk detecting unit140 may determine whether or not the vehicle is currently positioned inthe crosswalk using current position information of the vehicle decidedthrough the position data unit 181 and a crosswalk position of the mapdata pre-stored in the storing unit 110. Alternatively, for the purposeof more accurate decision, the crosswalk detecting unit 140 maydetermine whether or not the vehicle is currently positioned in thecrosswalk in consideration of all of the image data photographed in thecamera, the map data, and the position data.

The signal kind information determining unit 150 may determine signalkind information using an image data of a signal region portion of asignal lamp in the image data photographed in the camera. In detail, thesignal kind information determining unit 150 may determine an interestregion including the signal lamp in the image data using the vanishingpoint of the photographed image, and convert an image data of thedetermined interest region on the basis of a preset pixel value togenerate an interest region image data. Here, since the signal lamp ispositioned in an upper region of the vanishing point, the signal kindinformation determining unit 150 may decide that the upper region of thedecided vanishing point is the interest region.

Meanwhile, the signal kind information determining unit 150 may detectthe image data of the signal region portion of the signal lamp in theimage data of the decided interest region. In addition, the signal kindinformation determining unit 150 may determine the signal kindinformation on the basis of the image data of the signal region portion.

Here, the signal kind information, which is information for recognizingeach of a plurality of signals that may be expressed in the signal lamp,may include stop signal information, straight movement signalinformation, a left turn signal, a right turn signal, and the like.

The operated state determining unit 155 may determine an operated stateof the vehicle, for example, whether the vehicle is in a stopped state,in a driven state, or a parked state. In detail, the operated statedetermining unit 155 may determine whether or not the vehicle is in thestopped state using the image data photographed in the camera. In moredetail, the operated state determining unit 155 may generate a grayimage data for the image data, and sequentially compare a plurality offrames included in the generated gray image data with each other in atime sequence to determine whether or not the vehicle is in the stoppedstate. However, the present invention is not limited thereto. That is,the operated state determining unit 155 may determine whether or not thevehicle is in the stopped state on the basis of a signal sensed in thesensing unit 190 and motion data obtained in the position data unit 181,or may determine the operated state of the vehicle on the basis of realtime speed information of the vehicle obtained using controller areanetwork (CAN) communication of the vehicle.

Meanwhile, the electronic apparatus 100 according to an exemplaryembodiment of the present invention may include the augmented realityproviding unit 160 providing an augmented reality view mode. Theaugmented reality providing unit 160 will be described in detail withreference to FIG. 2.

FIG. 2 is a block diagram illustrating an augmented reality providingunit 160 according to an exemplary embodiment of the present inventionin detail. Referring to FIG. 2, the augmented reality providing unit 160may include all or some of a calibration unit 161, a 3D space generatingunit 162, an object generating unit 163, and a mapping unit 164.

The calibration unit 161 may perform calibration for estimating cameraparameters corresponding to the camera from the photographed imagephotographed in the camera. Here, the camera parameters may beparameters configuring a camera matrix, which is information indicatinga relationship between a real space and a photograph.

The 3D space generating unit 162 may generate a virtual 3D space on thebasis of the photographed image photographed in the camera. In detail,the 3D space generating unit 162 may obtain depth information from animage photographed by the camera on the basis of the camera parametersestimated by the calibration unit 161, and generate the virtual 3D spaceon the basis of the obtained depth information and the photographedimage.

The object generating unit 163 may generate objects for guidance, forexample, a route guidance object, a lane change guidance object, a lanedeparture guidance object, a crosswalk object, a pedestrian guidanceobject, and the like, on the augmented reality. Here, the object may beimplemented by a 3D object, a texture image, an art line, or the like.

The mapping unit 164 may map the object generated in the objectgenerating unit 163 to the virtual 3D space generated in the 3D spacegenerating unit 162.

Meanwhile, the control unit 170 controls a general operation of theelectronic apparatus 100. In detail, the control unit 170 may controlall or some of the storing unit 110, the input unit 120, the output unit130, the crosswalk detecting unit 140, the signal kind informationdetermining unit 150, the operated state determining unit 155, theaugmented reality providing unit 160, the communicating unit 180, thesensing unit 190, and the power supply unit 195.

Particularly, the control unit 170 may control the object generatingunit 163 to generate an object indicating the detected crosswalk whenthe crosswalk is detected from the image data photographed in the cameraduring a period in which the vehicle is operated, and may perform acontrol to output the generated object through the augmented reality.

As an example, in the case in which the vehicle is in the stopped stateas a decision result of the operated state determining unit 155 and thecrosswalk is present in front of the vehicle as a decision result of thecrosswalk detecting unit 140, the control unit 170 may control theobject generating unit 163 to generate a first object indicating thecrosswalk. In addition, the control unit 170 may perform a control tooutput the generated object through the augmented reality. Here, thefirst object may be an object for allowing the driver to recognize thatthe crosswalk is present in front of the vehicle.

As another example, in the case in which the vehicle is in the stoppedstate as a decision result of the operated state determining unit 155, asignal is a stop signal as a decision result of the signal kindinformation determining unit 150, and the crosswalk is present in frontof the vehicle as a decision result of the crosswalk detecting unit 140,the control unit 170 may control the object generating unit 163 togenerate the first object indicating the crosswalk during a period inwhich the vehicle is maintained in the stopped state. In addition, thecontrol unit 170 may perform a control to output the generated firstobject through the augmented reality. Here, the first object may be anobject for allowing the driver to recognize that the crosswalk ispresent in front of the vehicle.

However, in the case in which the vehicle moves in the stopped state inthe stop signal, the control unit 170 may control the object generatingunit 163 to generate a second object. In addition, the control unit 170may perform a control to output the generated second object through theaugmented reality. Here, the second object may be an object for warningthe driver that the crosswalk is present in front of the vehicle.

Here, the first object for allowing the driver to recognize that thecrosswalk is present in front of the vehicle and the second object forwarning the driver that the crosswalk is present in front of the vehiclemay be distinguished from each other.

In detail, the first and second objects may be distinguished from eachother by different colors. For example, the first object may beimplemented by a white color similar to a color of the crosswalk of areal world and be expressed on an augmented reality screen, and thesecond object may be implemented by a color for allowing the driver torecognize a danger state, for example, a red color, and be expressed onthe augmented reality screen.

In addition, the first and second objects may be implemented in a formin which they include an alpha channel related to transparency ofcolors, for example, RGBA (Red, Green, Blue, Alpha). In this case, thefirst and second objects may include transparent regions depending onthe alpha channel. That is, the first and second objects may beimplemented to include regions expressing colors and transparent regionsso as to correspond to the crosswalk of the real world.

In addition, expression positions of the first and second objects may beregions in which the crosswalk is positioned on the augmented reality.For example, when the crosswalk detecting unit 140 detects the crosswalkfrom the image data photographed in the camera, the control unit 170 maycontrol the mapping unit 164 to express the first and second objects atpositions in the augmented reality screen corresponding to the positionsat which the crosswalk is detected. According to an exemplary embodimentof the present invention described above, the object indicating thecrosswalk may be expressed as if it is positioned on the crosswalk ofthe augmented reality screen. Therefore, guidance may be provided to thedriver by a more intuitive method.

Meanwhile, according to an exemplary embodiment of the presentinvention, it may be determined whether or not a pedestrian is presenton the crosswalk, and notification corresponding to whether or not thepedestrian is present on the crosswalk may be performed.

In detail, the control unit 170 may determine whether or not thepedestrian is present on the crosswalk using the photographed imagedata, and control the object generating unit 163 to generate a thirdobject indicating whether or not the pedestrian is present. In addition,the control unit 170 may perform a control to output the generated thirdobject through the augmented reality. Here, the third object may be anobject for allowing the driver to recognize that the pedestrian ispresent on the crosswalk.

In addition, in the case in which a front vehicle of the vehicle startsin a state in which the pedestrian is present on the crosswalk, thecontrol unit 170 may perform a control so as not to perform the frontvehicle start guidance. That is, when the front vehicle start guidanceis performed in the state in which the pedestrian is present on thecrosswalk, collision between the vehicle and the pedestrian may occur.Therefore, in this case, the control unit 170 may perform a control soas not to perform the front vehicle start guidance.

In addition, in the case in which the vehicle starts in the stoppedstate in the state in which the pedestrian is present on the crosswalk,the control unit 170 may perform a control to perform guidance forwarning the driver that the pedestrian is present on the crosswalk. Thatis, when the vehicle starts in the stopped state in the state in whichthe pedestrian is present on the crosswalk, collision between thevehicle and the pedestrian may occur. Therefore, in this case, thecontrol unit 170 may perform a control to perform guidance indicatingthat the vehicle should not start.

Meanwhile, although examples in which the guidance of the crosswalk, thepedestrian, and the like, is displayed in a form of an image on theaugmented reality screen have been described above, the presentinvention is not limited thereto. Therefore, according to anotherexemplary embodiment of the present invention, the control unit 170 maycontrol the audio output unit 133 to output the guidance as an audio.

FIG. 3 is a view for describing a system network connected to anelectronic apparatus according to an exemplary embodiment of the presentinvention. Referring to FIG. 3, the electronic apparatus 100 accordingto an exemplary embodiment of the present invention may be implementedas various apparatuses provided in the vehicle, such as a navigationapparatus, a black box, a smart phone, other vehicle augmented realityinterface providing apparatuses, or the like, and may be connected tovarious communication networks and other electronic devices 61 to 64.

In addition, the electronic apparatus 100 may interwork GPS modules witheach other depending on radio signals received from artificialsatellites 20 to calculate a current position and a current time.

The respective artificial satellites 20 may transmit L band frequenciesof which frequency bands are different from each other. The electronicapparatus 100 may calculate the current position on the basis of a timerequired for the L band frequencies transmitted by the respectiveartificial satellites 20 to arrive at the electronic apparatus 100.

Meanwhile, the electronic apparatus 100 may wirelessly access a network30 through an access control router (ACR) 40, a radio access station(RAS) 50, and the like, via the communicating unit 180. When theelectronic apparatus 100 accesses the network 30, the electronicapparatus 100 may be indirectly connected to other electronic devices 61and 62 accessing the network 30 to exchange data.

Meanwhile, the electronic apparatus 100 may also indirectly access thenetwork 30 through another device 63 having a communication function.For example, in the case in which a module that may access the network30 is not included in the electronic apparatus 100, the electronicapparatus 100 may communicate with another device 63 having thecommunication function through a short range communication module, orthe like.

FIG. 4 is a flow chart schematically illustrating a control method of anelectronic apparatus according to an exemplary embodiment of the presentinvention. Referring to FIG. 4, the electronic apparatus 100 may detectthe crosswalk from the image data photographed in the camera during aperiod in which the vehicle is operated (S101). Here, the camera may bea camera formed integrally with the electronic apparatus 100 mounted inthe vehicle and photographing the front of the vehicle or a separateblack box mounted toward the front of the vehicle.

Then, the electronic apparatus 100 may generate the object indicatingthe detected crosswalk (S102). Here, the object may be implemented by a3D object, a texture image, an art line, or the like.

Then, the electronic apparatus 100 may output the generated objectthrough the augmented reality (S103). Here, the outputting (S103) mayinclude: performing the calibration on the camera to calculate thecamera parameters; generating the virtual 3D space for the photographedimage of the camera on the basis of the camera parameters; andpositioning the generated object on the virtual 3D space.

Hereinafter, a control method of an electronic apparatus 100 will bedescribed in more detail with reference to FIGS. 5 to 7.

FIG. 5 is a flow chart illustrating a control method of an electronicapparatus according to an exemplary embodiment of the present inventionin detail. Referring to FIG. 5, the electronic apparatus 100 maydetermine whether or not the vehicle is in the stopped state (S201).Here, the decision of whether or not the vehicle is in the stopped statemay be performed using the above-mentioned operated state determiningunit 155.

In the case in which the vehicle is in the stopped state, the electronicapparatus 100 may detect the crosswalk from the image data photographedin the camera during the period in which the vehicle is operated (S202).Here, the detection of the crosswalk may be performed using theabove-mentioned crosswalk detecting unit 140.

When the crosswalk is detected, the electronic apparatus 100 maygenerate the first object for allowing the driver to recognize that thecrosswalk is positioned in front of the vehicle (S203).

Then, the electronic apparatus 100 may output the generated first objectthrough the augmented reality (S204). In this case, the control unit 170may control the mapping unit 164 so that the generated first object isexpressed adjacently to the vehicle on the augmented reality screen.

Therefore, the first object may be expressed adjacently to the front ofto the vehicle on a road region of the augmented reality screen, and thedriver may easily recognize that the crosswalk is present in thevicinity of the front of the vehicle.

FIG. 6 is a flow chart illustrating a control method of an electronicapparatus according to another exemplary embodiment of the presentinvention in detail. Referring to FIG. 6, the electronic apparatus 100may determine whether or not the vehicle is in the stopped state (S301).Here, the decision of whether or not the vehicle is in the stopped statemay be performed using the above-mentioned operated state determiningunit 155.

When it is determined that the vehicle is in the stopped state, theelectronic apparatus 100 may determine the signal kind information usingthe image data of the signal region portion of the signal lamp in theimage data (S302). Here, the decision of the signal kind information maybe performed using the above-mentioned signal kind informationdetermining unit 150.

When it is determined that the signal kind information is the stopsignal, the electronic apparatus 100 may detect the crosswalk from theimage data photographed in the camera during the period in which thevehicle is operated (S303). Here, the detection of the crosswalk may beperformed using the above-mentioned crosswalk detecting unit 140.

Then, when the vehicle is maintained in the stopped state in a state inwhich the signal kind information is the stop signal, the electronicapparatus 100 may generate the first object for allowing the driver torecognize that the crosswalk is positioned in front of the vehicle(S304).

Alternatively, when the vehicle starts in the state in which the signalkind information is the stop signal, the electronic apparatus 100 maygenerate the second object for warning the driver that the crosswalk ispositioned in front of the vehicle (S305).

Then, the electronic apparatus 100 may output the generated objectthrough the augmented reality (S306). Here, the generated first andsecond objects may be expressed in forms distinguished from each otherin order to provide different kinds of guidance to the driver.

Therefore, the driver may not only easily recognize that the crosswalkis present in the vicinity of the vehicle, but also easily recognizewhether or not the vehicle may start.

FIG. 7 is a flow chart illustrating a control method of an electronicapparatus according to still another exemplary embodiment of the presentinvention in detail. Referring to FIG. 7, the electronic apparatus 100may determine whether or not the vehicle is in the stopped state (S401).Here, the decision of whether or not the vehicle is in the stopped statemay be performed using the above-mentioned operated state determiningunit 155.

When it is determined that the vehicle is in the stopped state, theelectronic apparatus 100 may detect the crosswalk from the image dataphotographed in the camera during the period in which the vehicle isoperated (S402). Here, the detection of the crosswalk may be performedusing the above-mentioned crosswalk detecting unit 140.

When the crosswalk is detected, the electronic apparatus 100 maydetermine whether or not the pedestrian is present on the crosswalkusing the photographed image data (S403).

Then, the electronic apparatus 100 may generate the object indicatingwhether or not the pedestrian is present (S404).

Then, the electronic apparatus 100 may output the generated objectthrough the augmented reality (S405). Therefore, the driver may easilyrecognize that the pedestrian is walking in the vicinity of the vehicle.

Meanwhile, according to the present invention, the electronic apparatus100 may perform a control so as not to perform the front vehicle startguidance in the case in which the front vehicle of the vehicle starts inthe state in which the pedestrian is present on the crosswalk.

In addition, according to the present invention, the electronicapparatus 100 may perform a control to perform the guidance for warningthe driver that the pedestrian is present on the crosswalk in the casein which the vehicle starts in the stopped state in the state in whichthe pedestrian is present on the crosswalk.

FIGS. 8A and 8B are views illustrating augmented reality screensexpressing a crosswalk object according to an exemplary embodiment ofthe present invention. FIG. 8A is a view illustrating an augmentedreality screen in the case in which the vehicle is stopped behind thecrosswalk during a stop signal 810. Referring to FIG. 8A, the electronicapparatus 100 may generate a first object 801 indicating the crosswalkpositioned in front of the vehicle, and output the generated firstobject 801 through the augmented reality. Here, the first object 801 maybe expressed adjacently to the front of to the vehicle on a road regionof the augmented reality screen. Therefore, the driver may easilyrecognize that the crosswalk is present in the vicinity of the front ofthe vehicle.

Meanwhile, FIG. 8B is a view illustrating an augmented reality screen inthe case in which the vehicle moves in the stopped state during the stopsignal 810. Referring to FIG. 8B, the electronic apparatus 100 maygenerate a second object 802 for warning the driver that the crosswalkis present in front of the vehicle, and output the generated secondobject 802 through the augmented reality. Here, the second object 802may be expressed adjacently to the front of the vehicle on the roadregion of the augmented reality screen, and may be implemented by acolor different from that of the first object 801 so as to bedistinguished from the first object 801. Therefore, the driver mayeasily recognize that the vehicle should not currently start.

Meanwhile, the above-mentioned first and second objects may beimplemented by a texture image and be expressed through the augmentedreality. This will be described in detail with reference to FIG. 9.

Referring to FIG. 9, the first object 801 for allowing the driver torecognize that the crosswalk is present in front of the vehicle may beimplemented by a white color similar to a color of the crosswalk of thereal world, and the second object 802 for warning the driver that thecrosswalk is present in front of the vehicle may be implemented by acolor for allowing the driver to recognize a danger state, for example,a red color and be expressed on the augmented reality screen.

Here, the first object 801 and the second object 802 may be implementedto include regions 801-1 and 802-1 expressing a color so as tocorrespond to the crosswalk of the real world and transparent regions801-2 and 802-2. In this case, transparency of the transparent regions801-2 and 802-2 may be adjusted by changing an A value, which is analpha channel value of the RGBA (Red, Green, Blue, Alpha), as anexample. In an exemplary embodiment of the present invention, the alphachannel value may have a value between 0.0 (completely transparent) and1.0 (completely opaque). Although the RGBA value has been used as avalue for displaying the color in an exemplary embodiment of the presentinvention, HSLA (Hue, Saturation, Lightness, Alpha), or the like, whichis another color displaying unit for displaying the alpha channel valueindicating transparency, may also be used.

FIG. 10 is a view illustrating an augmented reality screen expressing anotification object of a pedestrian according to an exemplary embodimentof the present invention. Referring to FIG. 10, the electronic apparatus100 may generate a third object 1001 for guiding that a pedestrian 1002is present on the crosswalk in the case in which the pedestrian 1002 ispresent on the crosswalk, and perform a control to output the thirdobject 1001 through the augmented reality. Therefore, the driver mayeasily recognize that the vehicle should not currently start whilerecognizing that the pedestrian is currently walking in front of thevehicle.

FIG. 11 is a view illustrating an implementation in the case in which anavigation apparatus according to an exemplary embodiment of the presentinvention does not include a photographing unit. Referring to FIG. 11, anavigation apparatus 100 for a vehicle and a separately provided blackbox 200 for a vehicle may configure a system according to an exemplaryembodiment of the present invention using a wired/wireless communicationscheme.

The navigation apparatus 100 for a vehicle may include a display unit131 provided on a front surface of a navigation housing 191, anavigation manipulation key 121, and a navigation microphone 123.

The black box 200 for a vehicle may include a black box camera 222, ablack box microphone 224, and an attaching part 281.

FIG. 12 is a view illustrating an implementation in the case in which anavigation apparatus according to an exemplary embodiment of the presentinvention includes a photographing unit. Referring to FIG. 12, in thecase in which the navigation apparatus 100 includes the photographingunit 125, the user may mount the navigation apparatus 100 so that thephotographing unit 125 of the navigation apparatus 100 photographs thefront of the vehicle and the display unit of the navigation apparatus100 may be recognized by the user. Therefore, a system according to anexemplary embodiment of the present invention may be implemented.

FIG. 13 is a view illustrating an implementation using a head-up display(HUD) according to an exemplary embodiment of the present invention.Referring to FIG. 13, the HUD may display an augmented reality guidancescreen thereon through wired/wireless communication with other devices.

As an example, the augmented reality may be provided through the HUDusing a front glass of the vehicle, an image overlay using a separateimage output apparatus, or the like, and the augmented reality providingunit 160 may generate an interface image overlaid on the augmentedreality image or the glass as described above, or the like. Throughthis, an augmented reality navigation apparatus, a vehicle infortainmentsystem, or the like, may be implemented.

According to various exemplary embodiments of the present inventiondescribed above, the guidance information is dynamically expressed by anaugmented reality method in a section in which the crosswalk is present,thereby making it possible to provide effective guidance to the driver,cause an interest of the driver, and promote safe driving andconvenience of the driver for the vehicle.

In addition, according to various exemplary embodiments of the presentinvention, related guidance is performed depending on whether or not thepedestrian is present on the crosswalk, thereby making it possible topromote safety driving and convenience of the driver.

Meanwhile, the control methods according to various exemplaryembodiments of the present invention described above may be implementedas programs and be provided to servers or devices. Therefore, therespective apparatuses may access the servers or the devices in whichthe programs are stored to download the programs.

In addition, the control method according to various exemplaryembodiments of the present invention described above may be implementedas programs and be provided in a state in which it is stored in variousnon-transitory computer-readable media. The non-transitorycomputer-readable medium is not a medium that stores data therein for awhile, such as a register, a cache, a memory, or the like, but means amedium that semi-permanently stores data therein and is readable by adevice. In detail, various applications or programs described above maybe stored and provided in the non-transitory computer-readable mediumsuch as a compact disk (CD), a digital versatile disk (DVD), a harddisk, a Blu-ray disk, a universal serial bus (USB), a memory card, aread only memory (ROM), or the like.

Although exemplary embodiments of the present invention have beenillustrated and described hereinabove, the present invention is notlimited to the above-mentioned specific exemplary embodiments, but maybe variously modified by those skilled in the art to which the presentinvention pertains without departing from the scope and spirit of thepresent invention as disclosed in the accompanying claims. Thesemodifications should also be understood to fall within the scope of thepresent invention.

1. A control method of an electronic apparatus, comprising: determiningwhether or not a vehicle is in a stopped state, detecting a crosswalkfrom an image data photographed in a camera during a period in which thevehicle is in the stopped state; generating a first object for allowinga driver to recognize that the crosswalk is positioned in front of thevehicle is generated when the crosswalk is detected; determining amapping position of the generated first object on a virtualthree-dimensional (3D) space for a photographed image of the camera; anddisplaying the first object through augmented reality by mapping thefirst object to the virtual three-dimensional space based on thedetermined mapping position.
 2. The control method of claim 1, furthercomprising: performing calibration on the camera to calculate cameraparameters; and generating the virtual 3D space for a photographed imageof the camera on the basis of the camera parameters.
 3. The controlmethod of claim 3, wherein the generating the virtual 3D spacecomprising: obtaining depth information from the photographed image onthe basis of the camera parameters; and generating the virtual 3D spaceon the basis of the obtained depth information and the photographedimage.
 4. The control method of claim 1, wherein the determining whetheror not a vehicle is in a stopped state comprising: generating a grayimage data for the image data; and sequentially comparing a plurality offrames included in the generated gray image data with each other in atime sequence to determine whether or not the vehicle is in the stoppedstate.
 5. The control method of claim 1, wherein the detecting thecrosswalk determine whether or not the vehicle is currently positionedin the crosswalk in consideration of the image data and at least one ofmap data and location data.
 6. The control method of claim 1, whereinthe first object include regions expressing colors and transparentregions.
 7. The control method of claim 1, wherein the displayingcomprises displaying at least one of a route guidance object, a lanechange guidance object, a lane departure guidance object, a pedestrianguidance object through augmented reality.
 8. The control method ofclaim 7, wherein the object is implemented by at least one of a 3Dobject, a texture image and an art line.
 9. The control method of claim1, further comprising: determining signal type information using animage data of a signal region portion of a signal lamp in the imagedata, wherein the detecting the crosswalk detects the crosswalk from theimage data photographed in the camera during the period in which thesignal kind information is the stop signal.
 10. The control method ofclaim 9, further comprising: generating the first object when thevehicle is maintained in a stopped state in a state in which the signaltype information is a stop signal; and generating a second object forwarning the driver that the crosswalk is positioned in front of thevehicle when the vehicle starts in the state in which the signal typeinformation is the stop signal.
 11. An electronic apparatus comprising:a display displaying a screen; a controller determines whether or not avehicle is in a stopped state, detects a crosswalk from an image dataphotographed in a camera during a period in which the vehicle is in thestopped state, generates a first object for allowing a driver torecognize that the crosswalk is positioned in front of the vehicle isgenerated when the crosswalk is detected, determines a mapping positionof the generated first object on a virtual three-dimensional (3D) spacefor a photographed image of the camera and controls the display todisplay the first object through augmented reality by mapping the firstobject to the virtual three-dimensional space based on the determinedmapping position.
 12. The electronic apparatus of claim 11, wherein thecontroller performs calibration on the camera to calculate cameraparameters and controls the generator to generate the virtual 3D spacefor a photographed image of the camera on the basis of the cameraparameters.
 13. The electronic apparatus of claim 12, wherein thecontroller obtains depth information from the photographed image on thebasis of the camera parameters and generates the virtual 3D space on thebasis of the obtained depth information and the photographed image. 14.The electronic apparatus of claim 11, wherein the controller generatinga gray image data for the image data and sequentially compares aplurality of frames included in the generated gray image data with eachother in a time sequence to determine whether or not the vehicle is inthe stopped state.
 15. The electronic apparatus of claim 11, wherein thecontroller detects the crosswalk determine whether or not the vehicle iscurrently positioned in the crosswalk in consideration of the image dataand at least one of map data and location data.
 16. The electronicapparatus of claim 11, wherein the display displays at least one of aroute guidance object, a lane change guidance object, a lane departureguidance object, a pedestrian guidance object through augmented reality.17. The electronic apparatus of claim 16, wherein the object isimplemented by at least one of a 3D object, a texture image and an artline.
 18. The electronic apparatus of claim 11, wherein the controllerdetermines signal type information using an image data of a signalregion portion of a signal lamp in the image data and detects thecrosswalk detects the crosswalk from the image data photographed in thecamera during the period in which the signal kind information is thestop signal.
 19. The electronic apparatus of claim 18, wherein thecontroller generates the first object when the vehicle is maintained ina stopped state in a state in which the signal type information is astop signal and generates a second object for warning the driver thatthe crosswalk is positioned in front of the vehicle when the vehiclestarts in the state in which the signal type information is the stopsignal.
 20. A non-transitory computer readable storage medium containinginstructions, that when executed by one or more processors, cause theone or more processor to perform a method, the method comprising:determining whether or not a vehicle is in a stopped state, detecting acrosswalk from an image data photographed in a camera during a period inwhich the vehicle is in the stopped state; generating a first object forallowing a driver to recognize that the crosswalk is positioned in frontof the vehicle is generated when the crosswalk is detected; determininga mapping position of the generated first object on a virtualthree-dimensional (3D) space for a photographed image of the camera; anddisplaying the first object through augmented reality by mapping thefirst object to the virtual three-dimensional space based on thedetermined mapping position.